1. Field of the Invention
This invention relates to the quality assessment of communications systems.
2. Related Art
In the increasingly liberalised telecommunications market, differentiation by quality is an important factor. The widespread installation of coding equipment in recent years has created the need for a new generation of assessment techniques. Conventional analysis assumes that the system is linear and time-invariant, and characterises it on the basis of delay, frequency response, noise level and noise spectrum. Modern networks exhibit far more complex effects and contain elements such as speech switches and compressive codecs that are highly non-linear and time varying.
Two main techniques are used in the industry to characterise the subjective quality of networks and their components. In listening tests a panel of subjects hear a series of sound clips that have been passed through simulated network conditions. Conversational tests require pairs of subjects to communicate through several simulated telephone links. In both cases the subjects are commonly asked to vote on a five point scale, from `excellent` to `bad`. Averaging across all subjects produces a mean opinion score (MOS) for each condition that mirrors the subjective quality of the network while reducing the random errors that appear in subjective voting.
The need to use a number of subjects for this kind of assessment makes the techniques expensive and hinders or prevents their application in the `live` telephone network. It is therefore desirable to produce a tester that will automatically measure quality on the same scale, and that produces the same scores, within experimental error and subjective expectation, as the mean of several conventional subjective tests. As will be discussed later, assessment devices have been developed that can predict the listening quality of speech passing one way through the telephone network, and have been extended by using a variable speech level to estimate the conversational quality of a two way link.
The current state of research in this area is to perform a number of measurements on the communications system under test, such as echo, delay, or degradation of speech, and predict the system's quality from these measurements. However, these tests neglect the reactions of users to the system's behaviour, which in turn can influence the way the system performs.
Telecommunications companies have considerable experience in using human subjects to assess the subjective quality of a communications network. This knowledge has been built up through international groups, such as ETSI and the ITU: see, for example, Methods for subjective determination of transmission quality: ITU-T Recommendation P.800. Subjective assessment uses a panel of subjects, who vote on a number of candidate network conditions. Their votes are averaged and examined, and give information about the subjective quality of the networks. This method is valued because the scores are directly related to peoples' opinion of quality. In contrast, conventional engineering metrics such as signal-to-noise ratio do not in general correlate well with speech quality. The subjective tests relevant to the present invention may be classified as:
listening, where the subjects hear sections of speech that have been passed through test networks and vote on what they hear, and PA1 conversational, where two subjects talk to each other over candidate network connections and vote on the speech quality of each conversation. PA1 a store for storing a plurality of signals forming a conversation PA1 a receiver for receiving signals from one or more complementary devices, PA1 a comparison device for comparing signals received by the receiver with signals stored in the store; PA1 selection means responsive to the comparison device for selecting from the store a signal for transmission to the complementary device or devices; PA1 transmission means for transmitting a signal selected by the selection means to the complementary device or devices. PA1 subjective quality of speech; PA1 side tone (the deliberately-imposed electrical coupling normally provided between the mouthpiece and earpiece of a telephone) PA1 degradation of speech due to filtering and coding in each direction; PA1 difficulty in communication caused by network characteristics such as: PA1 echoes from the local and the far end of the connection, or any intermediate points; PA1 degradation due to the (time-varying) operation of echo cancellers; PA1 loss or gain in speech loudness, and the variation of this loss with time and speech level; PA1 clipping due to the imperfect operation of speech detectors, and the variation of clipping with speech level; PA1 freeze-out (also known as dropout) due to instantaneous overloading in certain channels--a particular example is observed when intervals between speech are used to carry data packets: the beginning of an utterance may be lost as the system completes transmission of a packet; PA1 noise, which may be due to electrical sources, or crosstalk with data or speech signals; PA1 inability of the system to handle "double-talk" (the transmission of speech in both directions simultaneously.
Different questions prompt the subjects to vote on different aspects of their perception of the network, such as effort or quality.
Techniques have been developed (Models for predicting transmission quality from objective measurements: ITU-T P series recommendations: Supplement 3) to estimate the conversational quality of a conventional, linear, network using classical signal processing metrics such as echo delay and level. These measures rely on analysis of the network using artificial signals--usually sine waves or noise bursts--which may not be passed by a communications network designed for speech.
Models of human perception allow a more sophisticated analysis of the quality of speech transmitted by the network. Key features of the human senses, such as masking and threshold of hearing, are used to deduce whether errors in transmission are audible, and estimate their subjectivity. Higher level `perceptual` processing takes further account of the amount and distribution of errors, and their coincidence with certain parts of speech. This computation allows the subjective mean opinion score that would be given by a series of subjective tests to be predicted. Such models are described in: ITU-T Recommendation P.861: "Objective quality measurement of telephone-band (300-3400 Hz) speech codecs", and International Patent Applications WO94/00922 and WO95/15035.
The methods of conversational assessment described above may be used to weight perceptually-motivated quality measures to derive a perceptually-based estimate of conversational score. A system is described in the present Applicant's co-pending International Patent application (having the same filing date and claiming priority from the same two applications as the present application), which varies vocal level until an equilibrium is achieved, estimating listening effort, and then uses a conversational weighting based on echo and delay to estimate the quality of the connection for conversational speech.
There are good reasons for using artificially-generated speech-like signals, rather than recordings of human talkers, for testing a network designed to carry speech. In particular, artificial speech can be constrained to contain precisely defined phonemes, and are more easily reproducible to allow comparison of results. Artificial speech suitable for conversational testing is described in International Patent applications WO94/00922 and WO95/01011.
A dynamic conversational tester could take account of the change in certain properties of speech in hostile conditions. A description of these properties is given in International Patent application WO96/06495.
At the start of a call during which network quality is to be assessed, knowledge of certain network characteristics, such as delay and echo, is unavailable. However, these characteristics affect the conversational pattern. Furthermore, a fixed conversational pattern does not take into account the human response to losses in the line, such as noise, delay or freeze-out (the loss of a signal, for example due to capacity problems in the network, or delays in response to the start of a signal), which may be random in nature.